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Abstract 



A simple procedure for managing the 
perfect and zero response problem encountered 
in converting test scores into measures is 
presented. It allows the test user to choose 
among two or three reasonable finite 
representations of these boundary scores to 
suit the particular situation at hand. A 
convenient approximation is given which is 
accurate in all but highly asymmetric 
situations. 
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The problem . • 

Those who actually use latent trait models in their research 
and in the solution of practical educational' problems are 
inevitably, confronted with the conundrum of perfect scores. This 
is particularly so in situations involving Mastery Learning and 
Mastery Testing where the goal of, say, an 80 per cent pass rate 
guarantees that many students will achieve the maximum possible 
test score. .For the sake of completeness, and as an essential to, 
the reporting of every individual's performance, a finite measure 
is needed to represent these perfect scorers. 

That no such finite 'measure can be estimated from the 
interaction of a perfect scorer^with a test was noted by Wright 
and Stone, 

"All we can do in those situations is to observe 
that the person who scored all incorrect or all 
correct is substantially below or above the 
operating level of the test they have taken. If 
we wish to estimate a finite measure for such a 
person, then we will hav= to find a test for them 
which is more appropriate .to their level of 

ability." r\ 

(Wright & Stone, 1979, p. 6) 

Whilst the application of of a more appropriate test is obviously 

' \ 
preferred in such a situation, it is often inconvenient. 
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A solution 

One alternative is to xntroduce the assumption that perfect 
scores can be represented by some distribution of measures over 
the latent trait and to rationalise a scaled score from this 
distribution. This, however, undoes one of the major advantages 
of latent trait measurement - the "sample-free" calibration of 
items and "test-free" measurement of persons (Wright & Masters, 
1981, p. 5) . 

Is there any other way to establish a reasonable estimate of 
how far a perfect scorer might be above the operating level of 
the test? We can answer this in a probabal i'st ic sense. Consider 
an L item test with a maximum score^of m, . for each item, and let 

1-1 

be the perfect score. Then, assuming local independence, the 

probability of a perfect score, Q M , is 

- f ■ 

L 

Q M = Pr(r=M) = n Prfx^ra,), 

where x, indicates the score on each item, and r is the score on 
the test. Thus, given any measure b one can always calculate the 
probability that a person with that measure b will-Score M. An 
example of the probability curve generated by this equation is . 
shown in Figure 1. The assumption being made is that the perfect 
scorers would fit on our latent trait if they were given more 
difficult items (ie. that it was reasonable to give them this 
sort of test in the first place). . j 

. ' 5 V \ 
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Insert Figure 1 about here 

This- approach does not' tell us where the measures that might 
go with perfect scores actually are on the latent trait, but it 
does provide us with information which we can use to represent; 
the occurence of perfect scores in a reasonable fashion. To this 
end several points on the probability curve are worth noting: 
k ...the ability at which the score character istc 

curves for a perfect score-M and for a one s less 
than perfect score M-l intersect 
b 5 ...the minimum abiLity at which a person is more 

likely to score M than anything else 
b.„ ...the ability at which a person is likely to make a 

score other than M only 10% of the time 
b ...the ability at which a person is likely to make a 

•95 

score other, than M only 5% of the time. 

These values can help us to address the perfect score problem. 
There would seem to be no circumstances under which a perfect 
scorer would be assigned a value below b K , for below that value a 
score of M-l is more likely. Of course the highest possible 
ability that could be given to a perfect scorer is infinity, but 
a more realistic maximum value might be indicated by the point at 
which a less than perfect" score would be observed with 
probability 0.10 or 0.05: thus b. 9 or b. 95 might be reasonable 



maximum 



values. A median estimate is provided by b. 5 : this is 
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the lowest point at which a person is more likely to score M than 
to make any other score. In the absence of further information 
about the requirements of the specific testing situation, it 
would seem that this median value, b 5 , might be the best choice. 

Application to the Partial Credit Model 

The Partial Credit model (Wright & Masters, 1962, p. 43) states 
that for an m, +1 category item, the probability of a person who 
scored r, with scaled score b r , getting x on item i, with step 
values d (1 , d| 2 , ... , d |m is 



i 

X 



exp £ (b r " d l| > . 

irO 

p - x = 0 , 1 , 2 , . . . ,m, 



mj k 



£exp E < b , " d «i > 

k-0 j= 0 



0 

where 1 (b r -d M ) = 0 . 

JrO T M 



For a test with L items, the maximum score M is ^ m . 

Now, for each interesting probability level Q, we want to find 
the ability level b M for which 

Q =nPr(x l =m, ) = n p Mlmi , 

that is 

expX)(b M -d|| ) r 

q =n— — / 

SexpDb.-d,, ) / 
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- Since the solution of this equation is not explicit, we solve 
it by iteration: first take logarithms 



th b »- d H ) - i>9 S ex p| {b «- d .i > - iogQ = ° 



1-1 j - 0 1-1 



which simplifies to 



Mb ~ D - Vlog Lexp2(b M -d } " logQ " ° 



i= 1 



k--o l r0 



L IM 

where D = £ z d,, . 

1-1 M 



The derivative of this equation with respect to b M is 
M -y?kP u (Wright & Masters, 1982, p. 87) / 



Thus if b£ is an initial approximation to b M , we may 
Newton's method to find a better approximation, b„ 1 



~il__r_ 0 _ ! S JL — - 



use 



b M = 



1-1 k*0 

/] 



where PjJ,,, is P Mlk evaluated at b£ . 
A very good approximation to b M is given by 



- b° = b r loglog(l/Q) + 1/(M-1) 

M M-1 



\ 
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To see ..why this approximation is reasonable, consider the case of 
an L item dichotomous test (ie..M=U with all items of difficulty 
0. 

The probability of a person, with ability b L getting each item 
right is ( 1 + exp(-b ) ) 1 so 

Q = ( 1 + exp(-b L ) ) L 

which from exp(x) * 1 + x 

is approximately 



( exp( exp(-b ) ) ) L 



so that 

L(exp( -b L )) - log(l/Q) 
or b L logL - loglog(l/Q). 

But ... b L ., = log(L-l) 

so logL = b M + log(L/(L-D) 

_ b L .,+ 1/(L-1). 



J 
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r 



Substituting this in the equation^ above gives the approximation 
b a b -loglog(l/Q) + l/(jL-l). 

The introduction of b u1 relaxes the requirement that all items be of 
difficulty 0. Since the two b's cancel the difficulty level of the 
iest, all that is required is that the items be of similar difficulty 

The "crossover ability b x is the ability at which 

. Pr(r=M)- = Pr(r=M-l) . 
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Now Pr(r=K) - [I 



I mi 1 

n exp £ (b-dj, ) 



1*1 hO 



n 1 exp 1 (b-d, ) 

Ii1 krO ISO 



EL 
< P blfm r ,) " (P b|m, > > 

.= 1 



m ( -1 \ L L 

H exp£(b-d )(|Lnexp(b-d |m| )) 



1=1 i 



,-_ li- 



ra 2 exp(£(b-d,, )) 

7 



Equating these two . 

l\ exp(b x -d, ) -S£ exp(b x" d i«i ) 



We divide to get 



1 = £(exp(b x -d 



I = 1 



k ■ 

exp(-b K )53 (ex P ( " d i-, )) " 1 



• =1 



so 



b x = log( £ exp(d, ))•«■ 
in particular, when all items are dichotomous and of zero 



difficulty 



\ 



fc» ■ log L 
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Some examples 



(i) For the L=3 0 example of Figure 1 : 
'.. b; 2.20 
b„ = 2.30 



b. 5 - 2.63 



2.68 
4.56 
b„> 5.28 

This shows that the approximation works for the circumstances 



b. = 4.55 b u 9 



b 95 = 5.27 



from which it was dferived. 

(ii) For. a group of items from a blind persons' activity 
scale (Schulz et all 1982), 
with step difficulties: 



4 „ 5 

Stepl -,0.37 -2.91 
Step2 / - . -0.01 2.35 \ 1.58 



Items ^1 2 3 \ 

-0.8S -2.47 -0.47 3.35 



The maximium score for these 16 steps is 
values give the f ollowing\boundary abilities 



7 ' 8 9 10 

-4.05 -1.36 1.11 -0.20 
-0.47 1.99 - 2.80 

16. These step difficulty 



b„ =4.06 
b x = 4.30 
b 5 =4.59 
b. 9 = 6.54 
b. 95 = 7.26 



\ 



bj = 4.49 
b° 9 = 6.38 

b.9° s = 7 -. 10 
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: (iii) An example of items with, differing difficulty spreads 
20 items uniformly spread over 





-3.9 to 3.7 


r'1.9 to 1.9 


K 


4.60 


3.48 

» 


b x 1 


4.81 


3.59 ' 

i 




5.11 


3.92 


K 


5.02 


3.90 



(The* approximation suffers a little with greater spread, 
(iy) An example that is highly asymmetric 

19 items spread uniformly from -3.9 to -0.3 



i 
\ 

and one item at 3.9 



b 19 = 2.77 
bv = 3.98 



'19 I 



b. 8 = 4.04 
b. 5 ° '- 3.19 



In this asymmetric case the approximation is poor, f 
Zero scores 

•the analysis" above can be applied to zero scores with a 
slight adjustment: 

Newton's method becomes 

L _ 

k 



/ 



exp£(b -d n )\ - logQ 



= b — 

I i 1 kz.0 

i the approximation b° is . 

b° = b, + loglog(l/Q) - 1/(M-1) , 

0 1 

the cross-over point is 

12 

© . - •: • ■ ■ .- ) ■ / x 
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b x = -log( V%xp(-d t1 ) ) 



which, for the case where all items are dichotomous and have the 
same difficulty becomes "* -' 



b x . = -log L 



The case of zero and perfect items 

The same method can be applied to manage the perfect and zero 
items which occasionally arise during item calibration The 

resuitant-drfTiculty values would be interpreted in the same way 
as-^the ability values above. , \ \ 



Conclusion / 

v '• • "~ 

Although "there can be no substitute \f or the aquisition of 
further information regarding perfect scorers, it is often,, 
impractical to do so. The procedure described above uses the 
already-known features of the test ;(ie. item difficulties) to 
provide a readi/y interpreted measure for a perfect scorer. When 
this is applied to the Partial Credit model an iterative solution 
is required; however an approximation is available- which gives 
excellent results .^Generalization to zero scores and perfect and 
zero items is direct. 
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Figure 1. Score Characteristic Curves for a test of 10 dichotpmous 
V items following the simple logistic model where all items 

\ have zero difficulty. 
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